Skip to content

feat: Add file-based dashboard provisioner#1962

Open
ZeynelKoca wants to merge 1 commit intohyperdxio:mainfrom
ZeynelKoca:feature/dashboard-provisioner
Open

feat: Add file-based dashboard provisioner#1962
ZeynelKoca wants to merge 1 commit intohyperdxio:mainfrom
ZeynelKoca:feature/dashboard-provisioner

Conversation

@ZeynelKoca
Copy link

@ZeynelKoca ZeynelKoca commented Mar 22, 2026

Summary

Add a file-based dashboard provisioner that watches a directory for .json files and upserts dashboards into MongoDB (matched by name for idempotency).

Enabled by setting DASHBOARD_PROVISIONER_DIR to a directory path. Syncs every 30s by default (configurable via DASHBOARD_PROVISIONER_INTERVAL in ms). When the env var is not set, the provisioner is a no-op. This enables GitOps workflows where dashboards are managed as code, similar to Grafana's file-based provisioning (whose implementation heavily inspired this PR).

Provisioned dashboards are flagged with provisioned: true so they never overwrite user-created dashboards with the same name. Removing a file from the directory does not delete the dashboard (safe by default, same as Grafana).

Variable Required Default Description
DASHBOARD_PROVISIONER_DIR Yes Directory to watch for .json files
DASHBOARD_PROVISIONER_INTERVAL No 30000 Sync interval in ms
DASHBOARD_PROVISIONER_TEAM_ID No* Scope to a specific team ID
DASHBOARD_PROVISIONER_ALL_TEAMS No* false Set to true to provision to all teams

*One of DASHBOARD_PROVISIONER_TEAM_ID or DASHBOARD_PROVISIONER_ALL_TEAMS=true is required.

How to test locally or on Vercel

  1. Create a directory with a dashboard JSON file:
    mkdir /tmp/dashboards
    echo '{"name":"Test Dashboard","tiles":[]}' > /tmp/dashboards/test.json
  2. Start the API with DASHBOARD_PROVISIONER_DIR=/tmp/dashboards DASHBOARD_PROVISIONER_ALL_TEAMS=true
  3. Log in to create a team, wait ~30s
  4. Verify the dashboard appears in the UI
  5. Modify the JSON file, wait ~30s, verify the dashboard updates

References

@changeset-bot
Copy link

changeset-bot bot commented Mar 22, 2026

🦋 Changeset detected

Latest commit: ffe6158

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/api Minor
@hyperdx/app Minor
@hyperdx/otel-collector Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel
Copy link

vercel bot commented Mar 22, 2026

@ZeynelKoca is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 22, 2026

PR Review

  • ⚠️ No guard against user deletion/editing of provisioned dashboards via APIdeleteDashboard and updateDashboard in controllers/dashboard.ts have no check for provisioned: true. Users can delete or modify provisioned dashboards; they'll be silently restored/overwritten on next sync. Consider returning a 403 or warning when operating on provisioned dashboards, or at minimum document this as expected behavior.

  • ⚠️ provisioned field exposed in API responses — the new provisioned: boolean field will appear in all dashboard API responses (including external API v2) without any changes to response serialization. If this is intentional, it's fine; otherwise, consider stripping it in output schemas.

  • ⚠️ Coexisting duplicate names not reflected in unique index — the partial unique index (partialFilterExpression: { provisioned: true }) only enforces uniqueness among provisioned dashboards. User-created dashboards have no uniqueness constraint. The syncDashboards logic creates a provisioned copy alongside a user-created one with the same name, meaning users will see two dashboards with identical names in the UI. This should be documented in user-facing docs (it's in AUTO_PROVISION.md but not surfaced in the UI).

  • ✅ Logic is otherwise clean: finally block correctly resets isSyncing on early returns, new: false correctly detects insert vs update, partial filter expression is backward-compatible with existing data, and test coverage is thorough.

@ZeynelKoca ZeynelKoca force-pushed the feature/dashboard-provisioner branch 17 times, most recently from 4c24c45 to abdafb3 Compare March 22, 2026 23:40
ZeynelKoca added a commit to ZeynelKoca/ClickStack-helm-charts that referenced this pull request Mar 22, 2026
k8s-sidecar watches ConfigMaps labeled "hyperdx.io/dashboard: true"
across all namespaces and writes dashboard JSON to a shared volume.
HyperDX reads and upserts them natively via file-based provisioner.

Requires hyperdxio/hyperdx#1962
@ZeynelKoca ZeynelKoca force-pushed the feature/dashboard-provisioner branch 4 times, most recently from bacded7 to 3e9be8a Compare March 23, 2026 11:18
@ZeynelKoca
Copy link
Author

PR Review

* ⚠️ **`updateDashboard` / `deleteDashboard` don't guard provisioned dashboards** → Users can modify or delete provisioned dashboards via the internal API (`findOneAndUpdate` and `findOneAndDelete` have no `provisioned: false` filter). Deletions are recoverable (next sync re-creates), but updates will be silently overwritten. Consider adding a guard or at least a warning log in those controllers.

* ⚠️ **`stopDashboardProvisioner` resets `isSyncing = false` unconditionally** → If a sync is in-flight during server shutdown, clearing `isSyncing` immediately allows the finally block to race with the shutting-down DB connection. Low probability in practice, but could produce noisy errors on graceful shutdown. Consider waiting for the in-flight sync or not resetting `isSyncing` in stop.

* ℹ️ **`$setOnInsert` redundancy** → `name`, `team`, and `provisioned: true` are already included by MongoDB from the filter on upsert insert; the explicit `$setOnInsert` block is redundant. Not a bug, just noise.

✅ Schema validation (DashboardWithoutIdSchema) correctly strips provisioned so users can't set it via create/update API endpoints. Index design (partialFilterExpression: { provisioned: true }) is correct. Test coverage is solid.

  1. API guards for provisioned dashboards: Out of scope for this PR. Modifying the existing dashboard controllers changes the API contract and should be a separate PR. Deletions are self-healing (next sync recreates), and updates are overwritten on next sync. The GitOps source of truth is the file, not the database.
  2. isSyncing in stop: The sync completes in milliseconds (a few MongoDB upserts). The 10-second graceful shutdown window is more than sufficient
  3. $setOnInsert redundancy: Intentional. Explicit is better than relying on implicit MongoDB behavior

@ZeynelKoca ZeynelKoca force-pushed the feature/dashboard-provisioner branch from 3e9be8a to ffe6158 Compare March 23, 2026 12:43
@ZeynelKoca ZeynelKoca changed the title Add file-based dashboard provisioner featu: Add file-based dashboard provisioner Mar 23, 2026
@ZeynelKoca ZeynelKoca changed the title featu: Add file-based dashboard provisioner feat: Add file-based dashboard provisioner Mar 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant